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AN  EVALUATION  OF  TEN  TECHNIQUES 
FOR  CHOOSING  INSTRUCTIONAL  MEDIA 

ABSTRACT 

This  report  provides  an  evaluation  of  10  fundamentally  different 
techniques  applicable  for  choosing  instructional  media  for  proposed  Navy 
training  programs. 

The  method  of  evaluation  used  involved  six  training  system  designers 
each  ol  whom  applied  the  chosen  techniques  to  a  sample  of  seven  representa¬ 
tive  Navy  training  tasks.  The  results  of  this  application  were  then  ex¬ 
amined  by  a  panel  of  experts  who  judged  the  appropriateness  of  the  media 
chosen  to  the  task's  training  requirements.  This  served  as  the  basis  for 
ranking  the  techniques  in  terms  of  their  usefulness  in  the  design  of  Navy 
training  programs. 

The  ratings  for  each  of  the  top  three  ranked  techniques  were 
essentially  similar  in  value.  Based  on  these  ratings,  no  clear  cut  superior* 
ity  could  be  ascribed  to  any  of  these  three  techniques.  Further,  none  of 
the  techniques  was  found  adequate  for  direct  application  to  the  Navy. 

However,  one  of  the  three,  the  TAEG  technique,  was  selected  as  the  logical 
choice  for  father  development.  Guidelines  for  its  modification  are 
presented. 
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SECTION  I 
INTRODUCTION 

Choosing  a  cost-effective  six  of  instructional  media>  for  a  military 
training  program  has  proved  to  be  a  difficult  task.  Existing  formal 
procedures  for  media  selection  have  been  essentially  ineffective.  For  one 
thing,  the  procedures  are  Inexact;  the  selection  criteria  are  too  general 
for  Identifying  specific  media  for  specific  training.  The  level  of  detail 
is  also  at  issue.  Some  procedures  are  too  simple  to  handle  the  selection 
problem;  most  are  too  complicated  and  cannot  be  adapted  easily  to  a  range 
of  military  needs.  Various  factors  contribute  substantially  to  the  com¬ 
plexity's  of  the  media  selection  effort.  The  more  prominent  among  these 
are  outlined  below. 

a.  An  intrinsic  difficulty  in  optimizing  the  media  mix  exists  due 
to  the  nature  of  military  jobs.  Most  people  agree  that  the  selection  of 
training  supports  is  most  effective  tAien  unique  media  attributes  can  be 
associated  with  specific  training  objectives  or  subject  matter  activities. 
However,  military  operations  demand  a  variety  o';  skills  and  knowledges  in 
job  performance.  Numerous  activities  are  conducted  simultaneously  or  inter¬ 
actively  such  that  it  is  difficult  to  identify  those  task  characteristics 
relevant  to  media  selection.  The  frequent ’requirement  for  manual  control 
activities  throughout  job  performance  further  compounds  the  difficulty  in 
analyzing  job  task  requirements.  Thus,  job  performance  may  involve  relative¬ 
ly  straightforward  procedural  and  discrete  acts  (positioning  controls,  com¬ 
municating),  perceptual-discriminative  acts  (Identifying,  monitoring,  antic¬ 
ipatory  responses),  and  perceptual-motor  acts  (graded  response  in  continuous 
interaction  with  stimulus  changes).  Emphasis  is  placed  on  the  integration 
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of  responses,  coordination  and  timing,  time  sharing,  decision-making  and 
judgment.  These  activities  are  not  easily  partitioned  and  grouped  for  train¬ 
ing.  The  inability  to  conveniently  identify  discrete  beginnings  and  endings 
for  behavioral  processes  hampers  the  selection  of  media  appropriate  to  given 
job  requirements. 

b.  It  is  difficult  to  correlate  the  employment  of  a  specific  in¬ 
structional  medium  with  success  in  task  learning.  While  a  medium  provides 
known  capabilities  that  can  be  exploited  in' training,  it  is  the  quality  of 
the  courseware  and  the  utilization  procedures  that  are  major  determiners 
of  the  success  of  the  training  effort. 

c.  The  proliferation  of  instructional  hardware  during  the  past 
decade  has  resulted  in  a  wide  range  of  new  equipments,  yet  unbiased  in¬ 
formation  is  lacking  for  evaluating  the  claims  made  for  these  innovations. 

d.  Cost  is  a  factor  in  choosing  media.  Not  only  do  competitive 
training  media  vary  in  Initial  and  operating  costs,  but  each  has  unique 
costs  and  requirements  for  facilities,  personnel,  and  supplies.  The  time 
required  to  achieve  the  objectives  of  training  also  varies  among  media. 

i 

Therefore,  the  cost  of  achieving  instructional  goals  varies  significantly 
from  medium  to  medium.  The  training  system  designer  must  not  only  specify 
a  training  system  capable  of  accomplishing  the  instructional  goals  but  must 
also  consider  these  decisions  in  terms  of  cost  and  lead  time  factors. 

e.  While  various  media  selection  techniques  have  been  described 
and  demonstrated,  there  is  little  evidence  any  of  these  have  been  experi¬ 
mentally  evaluated  or  compared.  No  formal  attempts  have  been  made  to 
establish  the  relative  reliability  and  validity  of  these  techniques  for 
selecting  instructional  media  for  proposed  training  systems.  The  training 
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system  designer  Is  not  provided  the  necessary  basis  for  determining  which 
technique(s)  is  suitable  for  his  needs. 

PURPOSE 

The  present  report  provides  an  organization  of  information  pertaining 
to  the  utility  of  media  selection  techniques  for  training  system  design. 

It  provides  an  evaluation  of  prominent  media  selection  techniques  and  a 
comparison  of  the  relative  value  of  these  techniques  for  the  Training  Analy- 

l 

sis  and  Evaluation  Group's  (TAEG's)1  use  with  Navy  training.  The  specific 
objectives  of  the  study  are: 

a.  Review  and  place  in  perspective  a  representative  sample  of  the 
published  formal  media  selection  techniques.  The  techniques  chosen  reflect 
fundamentally  different  approaches.  As  a  condition  for  selection,  each 
technique  was  required  to  produce  at  least  two  media  options  for  each  learn¬ 
ing  event.  The  techniques  ranged  from  simple  to  complex  in  terms  of  the 
number  of  variables  accounted  for,  the  precision  of  the  definiti;;'  of  terms, 
and  the  operations  performed  (factors  integrated).  They  differed  also  in 
the  way  the  analyses  were  performed,  varying  from  wholly  manual  procedures 
to  combinations  of  manual  and  automated  modes. 

Our  review  of  these  techniques  provides  a  description  of  each 
approach,  an  examination  of  its  logic,  and  an  assessment  of  'the  practicality 
of  the  procedures  for  use  in  training  system  design. 

b.  Evaluate  the  usefulness  of  the  techniques  sampled  and  de¬ 
termine  a  rank-order  priority  of  usefulness.  Ten  techniques  were  eAamined 

^The  Training  Analysis  and  Evaluation  Group  of  the  Naval  Training  Equipment 
Ceiter,  Orlando,  Florida,  is  a  multi-disciplinary  group  tasked  by  the 
Chief, of  Naval  Education  and  Training.  One  of  its  functions  is  to  prepare 
training  system  specifications  for  proposed  Navy  training  systems. 
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in  terms  of  their  power  to  aid  Navy  training  system  designers  in  optimizing 
media  choices.  A  description  of  each  technique  is  provided  in  appendix  A. 

One  of  the  techniques  chosen  for  evaluation  was  the  recently  de¬ 
veloped  Training  Effectiveness  and  Cost  Effectiveness  Prediction  (TEGEP) 
Model  developed  by  the  TAEG  (TAEG  Report  No.  1,  1972).  Another  technique 
chosen  required  the  training  system  designer  to  choose  media  using  only  his 
intuitive  judgment  and  experience,  without  the  aid  of  formal  media  selection 
logic.  This  "Non-System"  technique,  which  telies  on  the  expertise  of  the 
individual,  has  been  conventionally  employed  by  training  system  designers. 
This  intuitive  approach  was  added  to  the  nine  formal  techniques  to  serve 
as  a  control  to  determine  the  extent  to  which  a  given  formal  technique 
would  influence  the  selection  of  media  for  defined  training  segments.  The 
evaluation  of  the  chosen  techniques  was  undertaken  to  measure  the  usefulness 
of  these  techniques  in  dealing  with  a  broad  range  of  Navy  training  tasks. 

The  selection  of  a  representative  sample  of  tasks  for  use  in  the  evaluation 
was  based  on  two  sources:  (1)  19  Categories  of  Navy  tasks  defined  by 

Willis  and  Peterson  (1961)  and  (2)  the  ranking  of  these  task  categories 
by  their  frequency  of  occurrence  in  a  sample  of  critical  Navy  jobs  by 
Bernstein  and  Gonzales  (1971).  From  these  categories  and  priorities, 
seven  task  categories  were  assembled  which  represent,  in  our  opinion,  the 
most  common  activities  performed  across  a  range  of  Navy  jobs.  Training 
objectives  were  written  for  each  of  these  se'en  task  categories. 

c.  Provide  design  recommendations  for  improving  the  most 
promising  media  selection  approach  in  our  sample  of  10  techniques  for 
TAEG  operations. 
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ORGANIZATION  OF  THE  REPORT 

In  order  to  organize  and  discuss  thq  analytic  procedures  and  the  re¬ 
sults  of  the  evaluation,  three  major  sections  are  presented  in  addition  to 
this  introductory  section. 

Section  II  describes  the  specific  behavioral  objectives,  the  media 
selection  techniques,  and  the  experimental  procedures  used  in  making  the 
comparisons.  It  also  presents  the  qualifications  of  the  training  system 
designers  who  employed  the  various  techniques  and  of  the  experts  who  rated 
the  output  of  these  techniques. 

Section  III  presents  the  results  of  the  analyses  and  provides  the 
opinions  of  the  training  system  designers  about  the  acceptability  and 
utility  of  each  of  the  10  systems  and  of  the  time  required  to  leurn  and 
then  use  each  technique.  Also,  analyses  are  provided  which  indicate  the 
significant  relations  between  specific  media  selection  techniques  and 
specific  behavioral  objectives. 

Section  IV  presents  a  statement  of  the  usefulness  of  the  media 
selection  techniques  and  identifies  the  common  characteristics  found  in 
the  more  useful  approaches.  The  most  promising  technique  for  the  TAEG's 
use  is  identified  together  with  a  set  of  criteria  for  improving  this 
technique.  * 

In  addition,  two  appendices  are  provided.  Appendix  A  contains  a 
detailed  description  of  the  media  selection  techniques  selected  for 
study;  appendix  B  contains  the  instructions  used  in  briefing  the  training 
system  designers  prior  to  their  use  of  the  10  media  selection  techniques. 
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SECTION  II 
APPROACH 

The  method  of  evaluation  involved  the  direct  application  of  each  of 
the  chosen  media  selection  techniques  to  a  sample  of  representative  Navy 
training  tasks.  The  results  of  this  application  were  then  examined  by  a 
panel  of  experts  who,  judged  the  appropriateness  of  the  media  chosen  to  the 
task's  training  requirements.  This  served  as  the  basis  for  ranking  the 
techniques  in  terms  of  their  usefulness  to  the  TAEG's  purpose. 

The  procedures  employed  required  six  training  system  designers  to 
learn  and  then  use  10  different  media  selection  techniques.  Each  designer 
independently  applied  each  of  the  techniques  to  seven  standard  training 
tasks.  This  produced  a  total  of  420  sets  of  media  choices.  Examining  this 
ppol  of  media  choices,  two  experts  in  training  system  design  independently 
rated  each  media  choice  for  its  ability  to  meet  the  training  objectives. 

The  ratings  of  all  media  choices  for  each  of  the  10  techniques  were  then 
combined  and  the  relative  utility  of  each  technique  for  media  selection  was 
determined. 

The  materials  developed  in  support  of  the  study,  the  personnel  in¬ 
volved  in  the  study,  and  the  procedures  employed  in  the  evaluation  are 
described  next.  . 

MATER IALS 

The  following  materials  were  prepared  for  use  in  the  evaluation: 
a.  Training  Objectives.  Based  on  the  Bernstein  and  Gonzales 
(1971)  ranking  of  common  Navy  task  categories,  the  top  seven  categories 
were  selected  and  modified  for  use  in  this  study.  In  the  order  of  their 
significance,  these  categories  are: 
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(1)  Recalling  facts  and  principles 

(2)  Recalling  procedures 

(3)  Non-verbal  identification  (classification) 

(4)  Detection  (non-verbal  uses) 

(5)  Using  principles,  interpreting,  inferring 

(6)  Decision-making 

(7)  Static  reaction  and  continuous  movement  / 

For  each  of  these  task  categories,  a  sample'  training  objective  was  prepared. 
A  list  of  these  objectives  is  shown  in  table  1. 

b.  Media  Selection  Techniques.  Ten  media  selection  techniques 
were  chosen  for  this  evaluation.  Five  criteria  were  established  for  deter¬ 
mining  the  media  selection  techniques  to  be  evaluated: 

(1)  Techniques  that  had  been  used  in  major  training  system 
design  projects  were  considered  as  prime  candidates  for  the  study. 

(2)  The  output  of  a  given  technique  had  to  provide  two  or  more 
useful  media  alternatives  for  a  given  training  objective(s),  not  just  a 
single  best  choice. 

(3)  Representative  techniques  involving  military  and  civilian 
application  were  sought. 

(4)  Fully  developed  procedures,  rather  than  just  descriptions 
of  a  process  used  in  a  particular  application,  were  sought.  However,  rfiere 
the  description  was  adequate,  a  given  technique  was  considered  a  candidate. 

(5)  The  techniques  eventually  selected  should  represent  signi¬ 
ficantly  different  approaches  to  media  selection. 

Based  on  these  criteria,  an  exclusive  sample  of  10  media  selection 
techniques  was  chosen. 
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TABLE  1.  SAMPLE  TRAINING  OBJECTIVES  REPRESENTING  HIGH  PRIORITY 
NAVY  TASK  CATEGORIES 


TRAINING  OBJECTIVES 

1.  Upon  request,  student  will  write 
Ohms  Law  and  define  each  symbol 
correctly. 


2.  Without  coaching,  a  student  will 
properly  activate  and  calibrate 
an  Ohm  meter,  and  measure  the 
resistance  in  an  unmarked 
resistor  within  the  range  of 
2000  to  100,000  ohms  with  a  pre¬ 
cision  of  +  5  percent. 

j  3.  When  presented  with  10  pulse 

analyzer  photographs  of  differ¬ 
ent  search  radars,  four  of  which 
are  long  range  air  search  radars, 
the  student  will  correctly  iden- 
j  tify  all  the  long  range  air 

search  radars  within  10  seconds. 

I 

'  4.  When  presented  ?ith  random  one- 
!  second  pulses  of  a  5000  Hertz 

tone  emerging  slowly  from  a  white 
noise  background  the  student  will 
detect  the  tone  before  it  reaches 
a  signal-to-noise  differential 
of  15  DB,  80  percent  of  the  time. 


ENTERING  BEHAVIORS 

Student  can  write  English  with  ade¬ 
quate  skill  but  la  not  familiar 
with  the  formula  for  Ohms  Law  and 
cannot  provide  the  technical  defi¬ 
nitions  for  the  factors  in  the  law. 
He  is  not  familiar  with  the  formal 
concepts  "voltage,"  "amperage,"  or 
'resistance." 

Student  can  write  and  define  Ohms 
Law  and  can  solve  the  Ohms  Law 
equation  for  resistance  when  the 
other  variables  are  given.  He 
cannot  set  up  the  equipment  nor 
collect  the  necessary  data  from  a 
live  circuit. 

Student  can  identify  and  measure 
pulse  width,  pulse  length,  and  other 
elements  of  a  radar  signature  and 
can  recall  the  concepts  basic  to  any 
signature  configuration.  He  cannot 
Identify  specific  classes  of  sig¬ 
natures* 

Student  can  identify  the  obvious 
tone  with  a  signal-to-noise  differ¬ 
ential  of  30  DB.  He  is  physiolog¬ 
ically  capable  of  hearing  the  tone 
at  *a  differential  of  15  DB  but 
cannot  as  yet  detect  it  in  the  back¬ 
ground  noise. 
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TABLE  1.  SAMPLE  TRAINING  OBJECTIVES  REPRESENTING  HIGH  PRIORITY 
NAVY  TASK  CATEGORIES  (CONT'D) 


TRAINING  OBJECTIVES 

5.  When  presented  with  a  paper  and 
pencil  exercise  concerning  the 
bringing  of  a  Captain's  gig 
alongside  a  pier  in  a  two-knot 
cross  current,  the  student  cox¬ 
swain  shall  diagram  his  approach 
to  the  pier  and  Identify  the 
changes  in  power  setting  and 
rudder  he  would  make  to  correctly 
compensate  for  the  cross  current 
and  to  account  for  the  power  and 
turning  characteristics  of  the 
boat. 

6.  Clven  an  EW  mission  objective  to 
locate  and  analyze  the  long 
range  air  search  radars  on  a 
specified  Island  using  a  speci¬ 
fied  platform  and  equipment  set, 
plan  an  EW  mission  capable  of 
obtaining  the  required  informa¬ 
tion  without  being  detected  by 
the  long  range  radars. 

7.  While  manning  the  ship  control 
station  on  an  SSN  in  an  in- trim 
condition  submerged  at  80  feet 
with  10  knots,  a  ship  control 

.team  shall  execute  a  change  in 
depth  of  100  feet  in  two  minutes 
and  stabilize  at  ordered  depth 
for  two  minutes  with  excursions 

I  of  no  more  than  five  feet. 


ENTERING  BEHAVIORS 

Ihe  student  can  already  recall  the 
various  individual  principles 
required,  but  he  has  not  attempted 
to  apply  one  or  more  of  these 
principles  to  realistic  problems. 


Student  can  operate  the  EW  systems 
required  in  the  proposed  mission  and 
can  identify  the  expected  target 
signals.  He  has  the  .required  back¬ 
ground  in  EW  system  management,  but 
he  has  not  seen  mission  plan  formats 
nor  has  he  prepared  or  evaluated 
mission  plans. 


Ihe  team  consists  of  an  experienced 
diving  officer  and  Ballast  Control 
Panel  Operator  with  trainee  bow  and 
stern  plane  operators  that  have  not 
reached  the  required  proficiency 
level.  Excursions  are  in  excess  of 
five  feet. 
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The  techniques  chosen  are  listed  in  table  2  with  brie£  statements 
concerning  the  purpose  of  their  development  and  where  applied.  Move 
detailed  descriptive  information  on  each  technique  is  given  in  appendix  A. 

As  indicated  earlier,  one  of  the  techniques,  the  Non-System  approach, 
was  added  to  represent  tha  conventional  approach  to  media  selection.  In 
this  case  the  training  system  designer  relies  on  his  skill,  past  experience, 
and  personal  preference  in  selecting  the  media  for  a  program  without  resort¬ 
ing  to  any  formal  procedure.  Tills  approach  was  included  as  a  control  to 
determine  to  what  extent  the  use  of  a  formal  technique  makes  a  difference  in 
the  choice  of  media. 

Another  of  the  techniques,  the  TAEG  approach,  was  included  to  determine 
the  usefulness  of  this  locally  developed  prototype  system  in  comparison  to 
other  formal  systems. 

The  nine  formal  techniques  differed  substantially  in  the  number  of 
and  kind  of  variables  considered  important  to  the  media  selection  process, 

A  total  of  14  variables  were  identified,  applicable  to  one  or  more  of  the 
techniques.  Table  3  lists  the  14  variables  and  identifies  (by  an  "X"  in  a 
cell)  which  variables  were  involved  in  a  given  media  selection  technique. 

In  addition,  the  last  column  of  table  3  shows  the  total  number  of  media 
candidates  available  in  each  technique.  The  variation  among  the  chosen 
techniques  is  considerable,  ranging  from  five  media  alternatives  in  one 
technique  to  31  alternatives  in  another.  In  one  case  the  media  candidates 
were  not  listed,  the  choice(s)  being  left  to  the  judgment  of  the  training 
specialist.  While  some  of  the  media  selection  techniques  used  similar 
variables,  each  technique  employed  a  different  combination  of  variables. 


taec  upon  ho.  a 


mu  2.  «DU  SELECTION  TECXMIQURS  USED  I*  THE  EVALUATION 


Mo. 

Author 

Spooaorlng  Organisation 

Purpose  of  Development 

Motes 

0 

"Noo-Systan" 

\ 

Traditional  approach.  The 
training  systan  designer 
used  his  eon  experience  and 
preferences  without  tha  aid 
of  a  formal  nadla  salac-  . 
tlon  technique. 

1 

"AP  Manual 
50-2" 

(31  Doe  U70) 

Department  of  Air  Porea; 
Air  Training  Coaamnd 

Part  of  a  guide  for  applying  tha 

Air  Porea  systana  approach  to  tha 
development  of  education  and  train¬ 
ing  programs;  to  ba  used  by  all 
personnel  who  plan,  develop, 
approve,  adalnlstar,  or  nanaga  Air 
Porea  instruction  and  Its  support¬ 
ing  sarvlcaa. 

Technique  was  developed  by 
Sr.  Va  H.  Allan  and  was 
originally  presented  In  a 
paper  "Basaarch  In  Instruc¬ 
tional  Media  and  Art  Educa¬ 
tion"  which  was  published 

In  plnal  Escort  of  tha  Uses 
of  Mawnr  Madia  In  Ait  Edu¬ 
cating  rroltct. 

2 

Brata 
(F«b  1971) 

Ilia  Band  Corporation 
(for  Headquarters,  USAPjf 

i 

Part  of  a  largo  Band  lnvaatlgatlon 
of  ayatana  for  Air  Pore*  Education 
and  Training;  to  be  used  in  design¬ 
ing  technical  training  program*  to 
Identify  appropriate  communication 
nadla. 

Concerns  talacoasnmlca- 
tlon  and  recorded  media. 

3 

Slagal  A 
Fadarnan 

(Jun  1970) 

Applied  Psychological 
Sarvlcaa,  Inc.  (for  tha 
Naval  Training  Device 
Cantar) 

To  recommend  a  haw  device  and 
claaaroon  aid*  to  ba  used  in  a 
proposed  training  prograa  for 
tha  Tactical  Coordinators  (TACCO) 

In  ASH  aircraft. 

Based  on  J.  P.  Guilford's 
"Structure  of  Intellect" 
noe«l. 

4 

TAEC  (1972) 

Training  Analysis  A 
Evaluation  Croup  of  tha 
Maval  Training  Equlp- 
mant  Cantar 

Por  usa  within  TAEC  In  choosing 
coat-affactlva  nadla  for  proposad 
training  systems  to  ba  used  In 
preparing  training  systaa  speci¬ 
fications. 

Partially  baaad  on  Nil 11a 

A  Peterson.  Deriving  Train- 
lng  Doric*  Implications 
fro*  Learning  Theory 
Principle*.  Vol.  I. 

5 

Brlggs(I970) 

American  Institutes  for 
Basaarch  and  Florida 

Stata  University 

For  us*  In  a  handbook  of  raaourca 
metarlala  and  atethods  for  teaching 
tha  systans  approach  to  tha  daalgn 
of  Instruction.  Oriented  to  tha 
teaching  of  academic  subjects  In 
clsssrooan. 

Uses  Gagne's  8  types  of 
learning  end  related 
conditions  of  laarnlng. 

6 

Armstrong, 
at  al. 

(Aug  1971) 

Bunker  Remo  (for  Air 

Porea  Human  Rsaourcas 
Laboratory) 

A  component  of  tha  systems 
approach  to  training  (SAT)  to  ba 
used  la  designing  training  for 
the  AP  A- 70  aircraft. 

Media  weights  within  spates 
were  designed  for  A-7D 
training,  and  cannot  ba 
considered  epproprlate  for 
othar  tasks. 

7 

Rhooa, 
at  *1. 

(May  1970) 

Waatlnghouaa  Learning 
Corp.  (for  Air  Fore* 

Hunan  Resources 
Laboratory) 

Initial  study  la  the  daalgn  of  an 
advanced  multi-media  Instructional 
systaa;  to  conp.ire  nadla  In  tarn* 
of  function,  flexibility  and  coat; 
for  usa  with  a  broad  rang*  of  Air 
Porea  training  tasks. 

Not  Intended  to  be  e  media 
selection  technique,  but 
provides  a  wealth  of  lnfor- 
nation  to  support  tha 
training  systan  daalgner 
la  choosing  nadla. 

8 

Walker 

(1947) 

Martin  Conpany 

A  general  method  for  us*  with  a 
broad  rang*  of  training  task*. 

Based  on  the  rating  of 
training- technique*  by  ex¬ 
perienced  training  person¬ 
nel.  Basic  approach  being 
uaad  by  Any  at  HumRRO, 

Ft  Knox,  KT. 

9 

Boucher, 
at  al, 

1971) 

Crusaun  Aaroapaca 

Corp  (for  Havy) 

Por  usa  la  selecting  nadla  for  tha 
Havy  F-1A  training  systan  and 
other  Havy  training  task*. 

/ 

Contain*  an  extensive  com¬ 
puter  data  bank  of  nadla, 
media  characteristics,  and 
costs,  with  automatic  sort¬ 
ing  to  list  all  nedla  with 
a  given  sat  of  character¬ 
istics. 

NOTE:  Source*  of  tb*«*  techniques  ar*  llicad  la  bibliography  and 
additional  du-rlpt.lv*  Information  la  locatmd  la  appendix  A. 


12 


TAHG  REPORT  NO.  8 


»iq»TT«AV 

8aAf3Bmaaxv  . 

®TP»H  jo  aaqomti  S  ^  5  3  N 


8J033BJ 

_ _ T80TJ08JJ  OldHTtlW 


jnaroaTnbaH 
*TPOH  Joj  PtoutaaoT 


uojaaaTJo 
_ QAaiipv  03  aoj! 


equanajjnbag 
SOj^pagg  03  £q  JAJ 3 1 8  ua  g 


»T8Xx»qy  o-paogoag 


^TITq^dao 
_  001^88  TlgOPPlTPOT 


*  83TlOTJ»ao8»gg 
_ _ goagpaa.j  quapnqg 


sapoH 

_ aagodsaa  quapnqg 


83^3  8^J  33  db jBug  sn-mtoTic 


_ _ 8ujujBaq  jo  a8aqg 


_  Xouanbajj  ijsaj, 


_ ^TOOTJJTQ  gsax 


8afjo8oqeg  ggai  Xuiojaa^  X  X 


/ 


I 

TAEG  REPORT  NO.  8 

/ 

This  Indicates  clearly  that  fundamentally  different  techniques  were 
compared. 

Separate  folders  were  prepared  for  each  of  the  media  selection 
techniques.  Each  folder  contained:  (1)  a  copy  of  the  original  document 
describing  the  technique,  and  (2)  step-by-step  procedures  developed  by 
TAEG  from  the  original  documentation  for  carrying  out  the  intent  of  the 
technique.  Identification  of  the  author (s)  and  the  sponsoring  organizations 
was  removed  from  the  folder  to  safeguard  against  influencing  the  evaluators. 

c.  Training  System  Designer  Rating  Scales.  Three  scales  were  pre¬ 
pared  for  the  training  systems  designers  to  use  in  recording  their  overall 
evaluation  of  the  utility  of  each  of  the  competitive  media  selection 
techniques.  The  first  scale  concerns  the  detail  in  the  media  prescription 
and  ranges  from  "broad  media  categories"  on  one  end  to  "specific  media 
categories  with  detailed  specification  data"  on  the  other  end.  The  Intent 
here  was  to  document  the  degree  of  detail  to  be  found  in  the  description  of 
a  proposed  type  of  media.  The  second  scale  concerns  the  level  of  confidence 
the  training  system  designer  has  in  the  validity  of  the  media  selected  and, 
therefore,  the  media  selection  technique.  It  ranges  from  "no  confidence" 

to  "high  confidence"  The  third  scale  concerns  the  training  system  designer's 
estimate  of  the  suitability  of  the  technique  for  use  in  TAEG  projects.  This 
is  a  general  measure  and  includes  ease  of  use,  time  to  use,  as  well  as  the 
previously  mentioned  factors  of  validity  and  detail  of  media  prescription. 
Scale  values  range  from  "under  no  circumstances  should  TAEG  use  this  system" 
to  "an  ideal  system  for  general  use  in  TAEG*"  Figure  1  depicts  these  scales. 

d.  Expert  Rating  Scale.  An  additional  rating  scale  was  developed 
for  use  by  the  expert  training  system  designers  in  rating  the  media  choices 
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Figure  l.  Scales  Used  by  Training  System  Designers  in  Rating  the  Utility  of  Media  Selection  Techniques  for  TAEG  Use. 
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made  employing  the  various  media  selection  techniques.  This  scale  ranged 
from  "mostly  poor  media  choices"  to  "comprehensive  set  of  media  alternatives." 
Figure  2  depicts  the  full  scale. 

PERSONNEL 

Two  groups  of  training  specialists  were  involved  in  the  evaluation. 

One  group,  designated  as  training  system  designers,  employed  each  of  the 
chosen  media  selection  techniques  in  deriving  an  optimum  media  mix  to 
achieve  the  training  objectives  for  each  of* the  seven  designated  tasks. 

The  other  group,  designated  as  experts  in  training  system  design,  evaluated 

/ 

and* then  formally  rated  the  media  choices  selected  by  the  training  system  " 
designers. 

TRAINING  SYSTEM  DESIGNERS.  Six  professional  members  of  the  TAEG  were 
assigned  as  training  system  designers  to  employ  the  chosen  media  selection 
techniques.  Three  were  psychologists,  two  were  educators,  and  one  was  an 
operations  research  analyst.  Except  for  one  of  the  education  specialists, 
all  had  previously  worked  as  members  of  training  system  design  teams.  None 
had  previously  used  any  of  the  chosen  media  selection  techniques. 

EXPERT  JUDGES.  Two  experts  in  training  system  design  were  selected  to  rate 
the  complete  set  of  media  choices  made  by  the  training  system  designers. 

One  was  Dr.  Alfred  F.  Smode,  formerly  the  executive  scientist  for  Dunlap 
and  Associates,  Inc.,  Darien,  Connecticut,  and  presently  the  senior  scientist 
of  TAEG.  Dr,  Smode  has  more  than  20  years  experience  in  Human  Factors  con¬ 
sulting  and  in  training  research  and  application.  The  second  expert  was 
John  D.  Armstrong,  who  at  the  time  was  the  senior  education  specialist  with 
TAEG.  Mr.  Armstrong  has  22  years  experience  as  an  advisor  to  Navy  schools 
on  the  use  of  training  equipment  and  has  recently  completed  a  Navy-wide 
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survey  on  the  products  and  services  required  to  support  Navy  training. 
PROCEDURES 

USE  OF  MEDIA  SELECTION  TECHNIQUES.  The  training  system  designers  were 
brle£ed  on  the  general  procedures  to  be  followed.  The  scenario  for  this 
briefing  is  provided  in  appendix  B.  Following  the  briefing)  the  training 
system  designers  first  employed  the  Non-System  approach;  then  each  of  the 
other  nine  media  selection  techniques  was  utilized.  The  order  in  which 
these  techniques  were  assigned  to  the  training  system  designers  was  varied 
to  insure  that  no  two  Individuals  would  apply  the  techniques  in  a  similar 
order. 

Before  applying  a  given  technique,  the  training  system  designer  studied 
the  folder  until  he  understood  the  process  and  recorded  the  time  required 
to  learn  the  technique.  He  then  proceeded  to  use  the  technique  to  choose 
media  for  each  of  the  training  objectives,  starting  with  objective  number 
one.  All  the  media  types  identified  as  useful  by  the  media  selection 
techniques  were  recorded.  After  usirig  a  media  selection  technique  with  all 
seven  objectives,  the  training  system  designer  recorded  the  time  required 
to  apply  the  given  technique  and  then  rated  the  technique  according  to  the 
three  scales  shown  in  figure  1. 

RATING  OF  MEDIA  CHOICES  BY  EXPERTS.  The  experts  independently  reviewed  the 
training  objectives  and  used  the  scale  shown  in  figure  2/ to  rate  each  of 
the  media  choices  made  by  the  training  system  designers.  The  ratings 
represented  the  experts’  estimate  of  the  usefulness  of  the  media  choices 
in  enabling  trainees  to  meet  the  specific  training  objectives.  Ilie' rating 
values  were  assigned  without  knowledge  of  which  training  system  designer 
had  made  the  media  choices  or  which  media  selection  technique  had  been  used. 
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SECTION  III 


RESULTS 


Three  types  o£  data  were  obtained  from  the  trials  of  the  media  se- 

✓ 

lection  techniques.  First*  the  training  system  designers  generated  420 
sets  of  media  choices  which  were  rated  by  two  experts  in  terms  of  their 
usefulness  in  achieving  specific  training  objectives.  Second,  each  train¬ 
ing  system  designer  rated  each  media  selection  technique  in  terms  of  his 
estimate  of  its  usefulness  in  the  design  of  Navy  training  programs.  And 
third,  each  training  system  designer  recorded  the  time  used  to  learn  each 
technique  and  the  time  required  to  apply  each  technique  to  seven  training 
objectives. 

Table  4  presents  the.  experts'  rating  of  the  420  sets  of  media  choices. 

It  also  presents  the  rank  order  correlations  (Rho)  between  the  ratings 
of  the  two  experts.  The  ratings  made  by  the  experts  varied  from  a  Rho  of 
.97  on  training  objective  7  to  a  Rho  of  -.60  on  training  objective  5. 

A  relatively  high  correlation  was  achieved  between  the  scores  assigned  by 
the  two  experts  on  four  of  the  objectives.  On  three  of  the  objectives  there 
was  one  lew  positive  and  two  negative  correlations.  In  these  instances, 
the  experts  disagreed  on  what  constituted  a  useful  solution.  While  this 
variation  can  in  part  be  attributable  to  ambiguity  in  the  training  objectives, 
it  also  highlights  the  feature  of  the  artificiality  of  the  rating  task.  The 
expert  ratings  were  made  with  difficulty  in  that  the  training  objectives 
were  described  out  of  context  to  any  larger  training  effort  and  many  of 
the  media  choices  were  described  with  insufficient  detail  to  insure  that 
the  experts  were  considering  and  rating  r.he  same  forms  of  media.  Part  of 
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the  variance  is  also  attributable  to  the  fact  that  the  experts  disagreed 
on  the  optimal  solution. 

Because  of  the  marked  differences  between  the  two  experts  in  the 
rating  of  the  media  choices  for  three  of  the  training  objectives,  the 
ratings  of  the  experts  were  combined  for  use  in  ranking  the  media  selection 
techniques.  Table  5  .summarises  the  ratings  made  by  the  experts.  The  sum 
and  mean  rating  made  by  each  and  their  combined  ratings  are  shown.  The 
combined  scores  provide  a  rank  order  of  the  various  media  selection 
techniques  in  terms  of  their  usefulness  to  the  Navy  as  broad  purpose  media 
selection  techniques. 

The  highest  ranked  technique  was  the  TAEG  technique  with  a  mean 
rating  of  3.60.  This  was  followed  closely  by  the  Non-System  techrique  with 
a  mean  rating  of  3.55;  then  Briggs  with  a  mean  rating  of  3.51;  Armstrong 
with  a  mean  rating  of  3.44;  and  AF  Manual  50-2  with  a  mean  rating  of  3.18. 

Table  6  shows  the  summary  of  an  overall  2-factorial  analysis  of  variance 
(ANOVA)  performed  on  the  data  in  table  4.  The  significant  interaction 
between  media  selection  technique  and  training  objectives  precludes  any 
interpretation  of  the  superiority  of  media  selection  techniques  independently 
from  the  types  of  learning  tasks  represented  by  the  training  objectives. 

The  training  system  designers  also  reported  the  time  required  ,to  read 
and  become  familiar  with  the  various  techniques.  The  average  tiAe  required 
per  technique  is  recorded  in  table  7.  This  time  varied  from  20.0  minutes 
for  technique  1  to  84,3  minutes  for  technique  3  (with  the  obvious  omission 
of  technique  0).  The  mean  time  for  all  attempts  to  learn  a  technique  was 
52,5  minutes.  The  formal  media  selection  techniques,  which  ranked  highest 
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TABLE  6.  ANOVA  SIMIARY  OF  EXPERTS'  RATINGS  OF  MEDIA  SELECTIONS 


SOURCE 

df 

MS 

F 

P  ' 

Technique  (A) 

9 

19.33 

13.81 

.01 

Training  Objective 
(B) 

6 

3.17 

2.26 

.05 

AB  Interaction 

5A 

4.04' 

2.89 

.01 

Within  Cells 

120 

1.40 

•* 

TOTAL 

839 

TABLE  7.  TIME  TO  LEARN  AM)  USE  MEDIA  SELECTION  TECHNIQUES 


Media 

Selection 

Technique 


Average  Time  to 
Learn  System 
(In  minutes) 


Ranks 


Average  Tima  per 
Media  Selection 
(in  minutes) 


Ranks 


0 


0 


11.2 


6 


1 

2 

3 

4 

5 

6 

7 

8 
9 


20.0 

50.7 

84.3 

35.7 
62.9 
68.6 
73.6 

25.3 
51. A 


3.5 

3.9 
33.1 
10.  A 

12.5 

17.8 

10.9 

12. A 

9.9 


1 

2 

10 

A 

8 

9 

5 

7 

3 


TAEG  REPORT  NO.  8 

in  the  previous  analysis,  required  35.7  ainutes  (technique  4)  and  62.9 
minutes  (technique  5). 

The  time  required  to  make  a  media  choice  with  the  various  media 

selection  techniques  also  varied.  The  mean  time  to  choose  a  set  of  media 

/ 

ranged  from  3.5  minutes  (technique  1)  to  33.1  (technique  2).  The  mean  tine 
for  all  attempts  to  make  a  media  choice  was  12.8  minutes.  The  media  selection 
techniques,  which  ranked  highest  in  the  previous  analysis,  required  10.4 
minutes  (technique  4),  11.2  minutes  (technique  0),  and  12.5  ainutes 
(technique  5),  per  media  choice.  It  should  be  noted  that  the  six  training 
system  designers  were  not  expert  in  the  use  of  any  of  these  techniques;  the 
data  reflects  their  initial  use  of  these  procedures. 

The  training  system  designers  also  rated  each  of  the  10  techniques 

\ 

according  to  the  three  scales  shown  in  figure  1  in  an  attempt  to  document 
their  estimates  of  the  utility  of  the  various  techniques.  The  means  of  the 
scores  assigned  by  the  six  designers  for  each  -of  the  10  techniques  is 
presented  in  figure  3. 

The  training  system  designers  ranked  the  TAfiG  technique  highest  on  all 
three  measures;  however,  the  ratings  never  exceeded  the  mld*range  on  the 
scales.  ' 
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SECTION  IV 


CONCLUSIONS  AND  RECOMMENDATION 

The  combined  rating  of  the  two  experts  for  each  of  the  three  highest 

rated  techniques  are  essentially  similar  in  value,  namely  3.60  (TAEG),  3.55 

/ 

(Non-System),  and  3.51  (Briggs).  Based  on  these  ratings,  no  case  can  be 
made  for  the  superiority  of  any  one  of  these  three  techniques  to  be  used  by 
TAEG  in  choosing  media  for  proposed  training  systems.  The  experimental 
evaluation  suggests  that  the  TAEG  technique  is  as  good  for  the  TAEG's  use  as 
any  of  the  other  techniques  evaluated.  Also,  none  of  the  techniques  were 
found  adequate  for  direct  application  by  the  TAEG.  Since  further  develop- 

I 

ment  of  any  of  the  techniques  would  be  required,  the  TAEG  technique  was 
selected  as  the  logical  choice  for  development  consonant  with  the  TAEG's 
needs. 

It  is  worth  noting  that  the  Non-System~ technique  was  fated  on.  slightly 
below  the  TAEG  technique  on  all  measures  of  usefulness.  This  traditional 
approach  can  only  be  expected  to  reflect  and  perpetuate  the  current  state 
of  the  art.  One  would  expect  considerable  variation  in  media  choices  since 
considerable  intuition  is  demanded  of  designers  in  using  the  Non-System 
approach.  There  is  also  no  potential  for  systematic  improvement  of  this 
technique.  /  • 

RECOMMENDATION  -  Expand  and  improve  the  existing  TAEG  technique  for  use  in 

/ 

TAEG's  media  selection  tasks.  Guidelines  for  this  development  effort  are 
presented  in  the  following  paragraphs.  Some  of  these  requirements  have 
already  been  either  partially  or  fully  met  by  the  TAEG  technique  or  other 
existing  techniques;  other  requirements  will  require  significant  development 
effort. 
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The  require*^  media  selection  technique  should  have  the  following 
characteristics: / 

a.  A  technique  for  professionals.  The  media  selection  technique 
should  be  designed  as  a  tool  to  assist  professional  education  technologists 
not  to  replace  them*  The  choosing  of  instructional  methods  and  media  for 
proposed  training  systems  remains  a  professional  task  that  cannot  he  fully 
procedural! red.  In  this  regard  the  technique  should  include  the  following: 

•  It  should  be  a  formal  process  with  a  set  of  terms  operationally 
defined. 

.  The  logic  for  media  selection  should  be  based  upon  theoretical 
concepts  about  the  processes  of  teaching  and  learning. 

.  Human  decisions  required  within  the  logic  should  involve  variables 
of  manageable  size  and  complexity.  Adequate  information  should 
be  available  to  provide  a  basis  for  the  decisions. 

.  The  major  task  of  the  media  selector  should  be  decision  making, 
with  data  retrieval  and  processing  a  minor  aspect  of  the  task, 
preferably  automated. 

•  It  can  be  expected  that  training  in  the  use  of  the  technique  will 
be  required  for  skillful  use.  In  turn,  the  technique  should  be 
useful  in  a  broad  variety  of  current  and  emerging  training 
situations. 

b.  Useful  Variables.  Variables  to  be  incorporated  into  the  media 
selection  logic  should  be  of  two  types.  The  first  type  includes  those 
variables  that  describe  a  training  program  as  a  learning  system,  i.e., 
stimulus  modes,  response  modes,  feedback  modes,  learning  strategies,  and 
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learning  styles.  The  second  type  Includes  those  variables  that  are  the 
major  determiners  of  the  cost  of  operating  a  training  system,  l.e.,  ac¬ 
quisition  and  operating  costs  and  use  rates. 

Specifically  the  variables  Include: 

.  Task  categories  with  associated  learning  guidelines. 

.  Stimulus  characteristics  of  the  tasks. 

.  Trainee  response  modes  required  by  the  guidelines. 

.  Trainee  feedback  requirements. 

.  Individual  learning  styles. 

.  Economic  analysis  based  on: 

-  life  cycle  costing 

-  use  rate  projections 

-  sensitivity  to  changing  requirements 

.  Practical  factors  such  as 

-  suitability  for  local  production 

-  stage  pf  training 

c.  Media  Alternatives.  Media  should  be  selected  from  an  extensive 
pool  of  alternatives.  While  none  of  the  tested  techniques  had  more  than 
31  media  typos,  approximately  100  generic  media  types  should  be  in  this 
pool.  This  would  cover  traditional  forms  o*f  instructional  media,  new  forms 
being  introduced  into  training  programs,  and  theoretical  configurations 
of  media  being  discussed  in  the  literature.  It  would  include  media  for 
supporting  classroom  instruction  and  individualized  instruction,  tele¬ 
communication,  and  on-the-job  training.  Each  medium  should  be  carefully 
defined  and  examples  provided.  Cost  data  for  the  acquisition  and  use  of  the 
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media  should  be  included  in  such  a  manner  that  tentative  cost  comparisons  of 
alternative  media  can  be  accomplished. 

d.  Training  System  Descriptions.  The  media  selection  technique 
should  support  the  preparation  of  descriptions  of  proposed  training  systems. 
Three  levels  of  descriptions  are  required: 

l 

.  Broad  instructional  strategies  for  general  types  of  learning 
tasks. 

.  Specific  mixes  of  media  for  specific  learning  tasks. 

.  Specific  design  and  use  parameters  in  such  detail  that  the 
descriptions  can  be  incorporated  into  training  system 
specifications  for  the  production  or  procurement  of  the 
media  packages. 

i 

e.  Growth  Potential.  The  system  should  be  capable  of  growth. 

This  Includes  a  capability  for  adding  or  changing  media  types,  learning 
guidelines,  and  economic  factors. 

f.  Interfaces.  The  media  selection  technique  should  interface 
smoothly  with  both  the  task  analysis  formats  dhd  training  system  specifi¬ 
cation  formats  to  be  used  in  the  TAEG. 

g.  Data  Manipulation.  Manual  computations  and  the  mechanics  of 
combining  factors  should  be  a  simple  task  for  the  training  system  designer. 
No  more  than  a  few  minutes  per  media  selection  should  be  required.  Some 
examples  of  useful  techniques  include: 

.  Two-dimension  tables 
.  Simple  math  models 
.  Automatic  data  processing 
.  Manual  card  sort  systems 
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h.  Cost  Model.  Preferably  the  coaputing  of  life  cycle  costs 
associated  with  the  media  selection  process  should  be  supported  with 
tables  of  cost  factors.  The  purpose  of  the  cost  model  is  media  selection 
not  the  development  of  budget  estimates.  Budget  estimates  are  to  be  made 
with  an  in-depth  economic  analysis.  Only  a  small  set  of  viaMe  training 
system  alternatives  should  be  subjected  to  an  economic  analysis. 

i.  Testing.  The  media  selection  technique  should  be  tested  via 

application  over  a  range  of  the  TAEG  projects.  Appropriate  modifications 

J 

and  further  development  should  be  accomplished  resulting  in  a  second  gener¬ 
ation  media  selection  technique  that  yields  optimum  results. 
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APPENDIX  A 

MEDIA  SELECTION  TECHNIQUES  INCLUDED  IN  THE  EVALUATION 

1.  Instructional  System  Development.  AF  Manual  50-2,  31  December  1970. 
Department  of  the  Air  Force. 

The  technique  is  presented  on  pages  5-13  and  5-14  under  the  heading 
"Selection  Considerations."  It  consists  of  a  simple  grid  type  table  with 
six  types  of  learning  objectives  across  the  top  and  10  types  of  instructional 
media  on  the  left  side.  Each  cell  formed  by  this  grid  is  classified  as  low, 
medium,  or  high  to  indicate  the  usefulness  of  the  medium  in  achieving  the 
learning  objectives.  The  table  was  developed  by  Dr.  William  H.  Allen, 
University  of  Southern  California,  and  published  in  the  Final  Report  of 
the  Uses  of  Newer  Media  in  Art  Education  Project.  NDEA  Project  No.  5-16-027, 
National  Art  Education  Association,  August  1966. 

2.  Bretz,  R.  The  Selection  of  Appropriate  Communication  Media  for  In¬ 
struction:  A  Guide  for  Designers  of  Air  Force  Technical  Training  Proizrams. 
R-601-PR.  February  1971.  Rand  Corporation,  Santa  Monica,  CA. 

Based  on  a  taxonomy  of  communication  media,  eight  classes  of  media 
are  identified.  They  are  audio-motion-visual,  audio- stil 1 -visual ,  audio¬ 
semimotion,  motion-visual ,  still-visual,  semimotion  ,  audio,  and  print. 
Specific  examples  of  each  class  are  listed!  In  addition,  a  series  of 
decision  flow  charts  with  decision  points  is  presented.  By  making  a  series 
of  "yes"  or  "no"  answers  to  questions  presented  at  these  decision  points, 
the  designer  is  routed  to  the  required  media  class.  All  the  media  in  this 
system  are  telecommunication  or  recorded  media. 

3.  Siegel,  A.  I.,  &  Federman,  P.  J.  Development  of  a  Method  for  Deriving 
Required  Training  Aids/Devices  and  Applications  to  the  Tactical  Coordinator 
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Position  in  ASW  Aircraft.  Technical  Report  NAVTRADEVCEN  68-C-0212-1. 

June  1970.  Naval  Training  Device  Center.  Orlando.  FI, 

In  this  system,  developed  by  Applied  Psychological  Services,  Inc., 
the  designer  is  required  to  use  a  scale,  "Amount  of  Intellectual  Operation," 
and  rate  five  intellectual  operations  in  a  task  or  specific  behavioral 
objective  (SBO).  These  five  operations  are  cognition,  memory,  divergent 
production,  convergent  production,  ^devaluation.  He  is  also  required  to 
use  a  scale,  "Taxation  Level  of  Intellectual  Activity,"  and  rate  each  of 
/  the  five  intellectual  operations  in  the  task  or  SBO,  These  factors  are 
combined  with  predetermined  values  of  the  suitability  for  each  instructional 
medium  to  exercise  each  of  the  intellectual  operations(values  derived  from 
the  mean  of  a  set  of  expert  opinions).  Numerical  values  that  result  from 
this  procedure  are  used  to  determine  if  the  training  objective  is  above  the 
difficulty  threshold  and  requires  media  support.  These  values  are  also  the 
basis  for  deriving  a  rank  ordered  set  of  media  appropriate  for  the  task. 
Sixteen  media  are  considered. 

4.  Staff  Study  on  Cost  and  Training  Effectiveness  of  Proposed  Training 
Systems.  TAEG  Report  No.  1.  1972.  Naval  Training  Equipment  Center, 

i 

Orlando,  FL. 

This  report  contains  a  two  phase  process  with  initial  selection  of 
media  based  on  task  categories  with  related  learning  guidelines  and  media 
capable  of  carrying  out  these  guidelines.  The  second  phase  involves  the 


1 
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use  o£  life-cycle  costing  of  a  small  set  of  useful  media.  In  the  experi¬ 
mental  study  being  reported,  only  the  first  phase  was  used.  In  addition, 
a  special  table  was  constructed  which  does  not  appear  in  TAEG  Report  No.  1 
but  is  based  on  the  20  page  media  selection  matrix  in  that  report.  It  con¬ 
tains  13  task  categories  along  the  left  side  and  20  media  alternatives 
across  the  top.  Each  cell  contained  the  description  "low,"  "medium,"  "high," 
or  "not  applicable"  according  to  ,its  usefulness  in  carrying  out  the  learning 
guidelines  associated  with  the  task  category. 

5.  Briggs,  L.  J.  Handbook  of  Procedures  for  the  Design  of  Instruction. 

1970.  American  Institutes  for  Research,  Pittsburgh,  PA.  p,  93. 

A  14-step  procedure  is  described.  Central  to  this  procedure  is  step 
six  which  requires  the  designer  to  classify  the  task  according  to  Gagne's 
eight  types  of  learning  and  then  note  Gagne'1  s  special  instructional  events 
or  conditions  of  learning  for  the  types  of  learning  to  be  considered. 
Seventeen  conditions  of  learning  are  identified  and  related  to  the  eight 
types  of  learning.  From  his  or,r  xperience,  the  designer  is  asked  to  choose 
media  that  meet  the  criteria  established  by  the  conditions  of  learning  and 
other  factors. 

6.  Armstrong,  G.,  et  al.  The  Systems  Approach  to  Training  (SAT)  Training 
Analysis  Guide.  August  1971.  Air  Force  Systems  Command,  Air  Force  Human 
Resources  Laboratory  (FT),  Williams  AFB,  Arizona. 

The  technique,  prepared  by  Bunker-Ramo,  Electronic  Systems  Division, 
is  pre#4nted  on  pages  43  to  62  under  headings,  "Media  Selections"  and 
"Application  of  the  Media  Selection  Technique."  It  involves  classifying 
the  tasks  according  to:  (a)  three  levels  of  learning,  (b)  four  classes 
of  activity,  (c)  four  complexities  of  cognitive  or  psycho-motor  behavior. 
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(d)  three  stimulus  forms,  and  (e)  three  response  aodes.  Five  aedia  options 
are  available,  i.e.,  the  operational  system,  classrooa,  learning  center, 
simulator,  and  part- task  trainer.  Media  analysts  have  coapiled  tables  of 
factors  to  indicate  the  usefulness  of  each  medium  in  supporting  each  of  the 
task  factors  lifted  above.  Media  selection  is  based  on  choosing  media  with 
relatively  high  overall  ratings. 

7.  Rhode,  W.  E. ,  et  al.  Analysis  and  Approach  to  the  Development  of  an 
A^ivanced  Multi-media  Instructional  System.  AFHRL-TR-69-30,  Vol.  1,  May  1970. 
Air  Force  Human  Resources  Laboratory,  Air  Force  Systems  Command,  Wrlght- 
Patterson  Air  Force  Base,  Ohio. 

Although  not  intended  as  a  media  selection  technique,  this  study, 

conducted  by  the  Westinghouse  Learning  Corporation  for  AFHRL,  contained 

•0 

tabulated  information  including  the  ranking  of  media  according  to  cost, 
flexibility,  potential  for  individualized  Instruction,  interaction  capacity, 
and  cost  sensitivity  to  changes  in  student  load.  A  simple  media  selection 
procedure  was  prepared  using  these  tables. 

8.  Walker,  R.  W.  An  Evaluation  of  Trainins  Methods  and  Their  Characteris- 

ft 

tics.  Human  Factors.  1967,  7,  pp.  347-354. 

A  sizeable  number  of  technical  training  specialists  and  supervisors 
were  asked  to  list  the  selection  factors  they  used  in  deciding  which  train¬ 
ing  techniques  to  use.  Thirty-four  factors  were  identified.  Sixteen 
training  techniques  were  also  listed.  Then  a  group  of  20  training  specialists 
rated  each  training  technique  in  terms  of  the  selection  factors.  A  Selection 
Criteria  Matrix  was  constructed  which  contained  the  mean  values  for  each 
training  technique  and  selection  factor  pair. 
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To  use  the  matrix,  the  designer  first  establishes  the  real-life 
parameters  that  must  be  met  in  a  training  program,  i.e.,  short  development 
time,  low  budget,  etc.  He  then  examines  the  matrix  to  find  the  training 
technique  that  best  meets  the  parameters. 

9.  Boucher,  B.  G. ,  Gottlieb,  M.  J.  &  Morganlander,  M.  L.  A  Selection 
System  and  Catalog  for  Instructional  Media  and  Devices.  October  1971. 

Grumman  Aerospace  Corporation,  Bethpage,  N.  Y. 

This  technique  was  developed  in  support  of  the  Navy  F-14  project. 

Media  selection  is  accomplished  through  the  use  of  a  Media  Capabilities 
Matrix.  Across  the  top  are  23  media  characteristics  listed  under  Presenta¬ 
tion,  Instructional  Strategy,  and  Student  Response.  Along  the  left  side  are 
29  media  types.  Each  cell  in  the  matrix  contains  one  of  11  symbols  indicating 
the  degree  to  which  the  medium  contains  the  media  characteristics  listed 
across  the  top  of  the  matrix.  To  use  the  matrix,  the  training  system 
designer  first  determines  which  of  the  23  media  characteristics  he  requires. 
Then  by  inspecting  the  matrix  he  determines  which  medium  best  meets  these 
requirements.  The  system  is  designed  for  automatic  data  processing.  The 
data  processing  can  also  be  done  by  hand. 
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APPENDIX  B 

BRIEFING  PLAN  FOR  MEDIA  SELECTORS 

One  of  the  reasons  the  TAEG  ties  established  was  to  develop  training 
systea  specifications.  These  training  specifications  incorporate  a  mix  of 
instructional  media.  The  aix  might  include  classrooms,  laboratories, 
simulators,  and  operational  systens.  The  instructional  aedla  selected 
should  be  both  adequate  to  accoaplish  the  training  and  also  be  a  sound  choice 
froa  the  economic  analysis  point  of  view. 

The  TABG  personnel  need  one  or  aore  techniques  for  use  in  selecting 
media  for  training  systea  specifications. 

Various  training  systea  designers  have  developed  techniques  for 
determining  which  instructional  media  to  incorporate  into  their  training 
systea  designs.  They  have  developed  these  media  selection  techniques 
because  choosing  a  training  medium  can  be  a  perplexing  problem.  The  de¬ 
signer  must  choose  among  hundreds  of  types  of  instructional  media,  each 
suited  in  varying  degrees  to  various  training  tasks. 

These  different  media  selection  techniques  have  been  described  in 
the  training  literature,  but  no  one  technique  seems  to  be  widely  recognized 
as  a  superior  approach  to  choosing  instructional  media. 

Your  task  is  to  use  some  of  these  media  selection  cechnlques--ones 
that  appear  to  be  useful  to  the  TAEG  operation.  You  will  record  media 
selections,  and  rate  each  technique  in  terms  of  its  potential  usefulness 
to  the  TAEG  in  designing  training  systems. 

You  will  use  nine  different  systems  and  one  non- system,  or  ten 
different  methods.  A  self-contained  package  of  directions,  forms,  and 
background  information  has  been  prepared  on  each  of  the  methods. 
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Each  package  contains  (1)  step-by-step  directions  for  selecting 
media*  and  (2)  background  information  on  the  media  selection  technique. 

You  will  also  be  supplied  with  a  set  of  training  objectives  and  a  worksheet. 
Each  of  these  items  will  now  be  described  in  greater  detail. 

A  set  of  training  objectives  is  provided  which  describes  seven  spe¬ 
cific  training  tasks.  This  same  set  of  training  objectives  will  be  used  in 
trying  out  each  of  the  media  selection  techniques.  Media  selected  for  the 
seven  training  objectives  will  be  recorded.' 

The  "Worksheet  for  Media  Selectors*"  figure  4*  provides  space  for  a 
media  selector  to  record  all  his  data  generated  during  the  trial  of  a  single 
media  selection  technique.  Each  media  selector*  therefore*  will  fill  out 
ten  of  these  forms,  one  for  each  technique. 

There  are  seven  items  on  the  worksheet:  Item  1  provides  space  to  log 
the1  number  of  minutes  required  to  learn  the  technique  prior  to  its  use  with 
, the  seven  objectives.  If,  during  the  use  of  the  technique,  additional  back¬ 
ground  study  is  required*  this  should  be  added  to  the  figure  originally 
logged  in  Item  1 . 

Item  2  provides  space  to  log  the  total  number  of  minutes  of  effort 
required  to  make  media  choices  for  all  seven  objectives,  exc luding  the 
time  logged  in  Item  1.  . 

Item  3  provides  space  to  log  the  set  of  useful  media  alternatives  for 
each  of  the  seven  training  objectives.  Two  or  more  of  the  most  useful 
media  options  identified  through  the  use  of  the  media  selection  technique 
should  be  logged  for  each  objective. 

Items  4*  5,  and  6  provide  spaces  to  log  a  number  between  0  and  100 
representing  your  rating  of  the  media  selection  technique  on  three  different 
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WORKSHEET  FOR  MEDIA  SELECTORS 

Media  Selection  Technique:  Name: 

’  *  ■» 

1.  Tine  required  to  learn  to  use  the  system:  _ minutes. 

2.  Time  required  to  prescribe  media  for  all  7  Training  Objectives: 

minutes. 

3.  Media  Selections  are: 

TRAINING  OBJECTIVES: 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

4.  Detail  of  Media  Prescription: 

5.  Confidence  in  Validity  of  Media  Selection  Process:  _ 

6.  Suitability  for  TAEG: 

7.  Comments:  _ 


Figure  4,  Sample  of  Worksheet  Used  for  Media  Selectors* 
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scale*.  These  scales  appear  on  the  second  page  o£  the  Worksheet  for  Media 
Selectors. 

Itea  7  provides  space  for  you  to  log  your  comments  on  the  technique- - 
any  strong  impressions  or  gripes  concerning  the  logic  or  lack  of  logic  in 
the  approach  to  media  selection  and  things  that  should  be  changed  to  make 
it  more  useful. 

Most  of  the  authors  of  media  selection  techniques  have  described  their 
processes  in  detail  but  have  not  prepared  step-by-step  procedures  for  others 
to  use  in  replicating  the  method  for  their  own  use.  Therefore,  step-by-step 
directions  were  prepared  for  use  in  this  experiment.  These  directions  do 
not  go  beyond  the  descriptions  prescribed  by  the  originators  of  the  systems. 
If  a  step  appears  vague  or  requires  a  judgment  for  which  little  background 
information  has  been  assembled,  it  is  probably  because  these  qualities  were 
in  the  original  descriptions. 

Background  information  is  provided.  This  is  material  prepared  by 
the  original  author  and  will  provide  the  rationale  and  descriptions  of  the 
techniques  needed  for  you  to  determine  the  level  of  confidence  you  have  in 
the  system.  Media  alternatives  are  frequently  described.  You  will  not  be 
able  to  use  a  system  without  first  making  a  careful  study  of  the  background 
information,  although  you  may  choose  not  ter  read  every  page.  Study  each 
system  until  you  understand  how  it  works. 

All  media  selectors  will  first  receive  the  non-system  package.  The 
task  is  to  select  what  you  intuitively  feel  are  the  best  media  alternatives 
without  the  aid  of  any  form  of  selection  technique.  Read  the  training 
objectives  and  from  your  experience  record  the  two  or  more  media  options 
you  believe  to  be  well  suited  to  the  training  task. 
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To  determine  the  order  In  which  you  will  use  the  remaining  seven 
packages,  select  a  piece  of  paper  from  the  box.  The  piece  of  paper 
will  indicate  which  of  the  random  sequences  you  will  follow. 

Use  one  folder  each  day  until  you  have  completed  all  the  folders. 

Pick  up  a  folder  each  morning  between  0800  and  0815  and  work  on  the  task 
described  in  the  folder  until  you  have  completed  the  task.  Upon  finish* 
ing  the  task,  immediately  return  the  folder  and  worksheet.  Work  independent¬ 
ly.  Do  not  receive  help  from  other  associates. 

If  you  have  a  problem  with  any  procedure  and  cannot  solve  it  without 
help,  discuss  the  problem  with  the  principal  investigator. 

If  you  must  make  assumptions  concerning  the  overall  training  system 
in  order  to  select  media  for  one  phase  of  it,  state  your  assumptions  on 
the  back  of  the  worksheec. 

Do  not  discuss  the  techniques  or  your  media  choices  with  others  until 
after  the  experiment  has  been  completed.  You  will  be  working  on  some  of 
the  packages  before  other  media  selectors,  and  we  must  not  have  your  work 
influence  their  media  selections. 
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